Runtime localized cooling of high-performance processors

ABSTRACT

A plurality of thermal electric cooler (TEC) elements are formed in a TEC grid structure. Control logic dynamically varies a supply current supplied to each TEC element (or group of TEC elements) in the TEC grid based on changes in power density respectively associated with areas cooled by each of the TEC elements or group of TEC elements.

STATEMENT REGARDING FEDERALLY SPONSORED RESEARCH OR DEVELOPMENT

This invention was made with Government support under the PathForwardProject with Lawrence Livermore National Security (Prime Contract No.DE-AC52-07NA27344, Subcontract No. B620717) awarded by DOE. TheGovernment has certain rights in this invention.

BACKGROUND Description of the Related Art

Modern high-performance processors can easily dissipate more than 100 Wof power. The heat generated by the processors varies both spatially andtemporally. Depending on the nature of the workload, different locationson the die can have a heat flux far greater than the spatial averageheat flux of the die. Furthermore, the heat flux can vary in time aswell. The hot-spots thus generated can result in localized spikes intemperature. However, the spatial average temperature across the die canremain below the maximum sustainable value. While the location oftemperature sensors is determined based on studies conducted offline,the temperature sensors are not always located close to the actualruntime hot-spots and thus may not accurately reflect hot-spottemperature.

It is well known that hot-spots reduce the lifetime-reliability of thedevice significantly. The other important side effect of hot-spots isthe need to design for the worst-case heat scenario. That results in notonly the local heat sink being overprovisioned, but causes the airconditioners in data centers to be designed to handle heat fluxes thatare rarely actually observed. Millions to billions of dollars are spenton the cooling infrastructure in data centers that is, more often thannot, underutilized.

Various hardware and software techniques have been implemented orproposed to manage temperature dynamically. Architectural andmicroarchitectural modifications such as instruction fetch throttlingattempts to reduce power consumption when a threshold temperature iscrossed. Reigning in dissipated power via Dynamic Voltage FrequencyScaling (DVFS) of the cores and/or memory units has been studiedextensively and is still an open research area. Thread scheduling andmigration between cores is a software technique aimed at maintaining abalanced thermal field. Techniques employing DVFS typically assumeeither spatial or temporally averaged temperatures and deal with worstcase thermal management. As noted earlier, even though the processortemperature sensor may show a particular temperature, e.g., 85° C.,localized hot-spot temperatures can be much higher. Current processorstry to maximize the thermal budget by increasing the core frequencieswhenever thermal headroom is available. These techniques heavily rely onthe heat sink's ability to remove the heat generated.

The system integrators are assigned the task of designing a heat sinkthat is cheap to manufacture and can reliably dissipate the heat. Thegeneral strategy, once again, is to design for the worst case. Existingcooling mechanisms such as heat spreaders, cold plates, heat pipes, andmicrochannel heat sinks are typically designed only for uniform heatextraction. Furthermore, in an effort to reduce manufacturing costs,some system integrators choose lower grade materials for the heat sink,which leads to reliability issues. Microfluidic cooling, whileattractive, requires a non-trivial amount of effort to etch the channelsand would also require significant changes in the packaging process.Thus, improved mechanisms to dissipate heat associated with operation ofintegrated circuits are desirable.

SUMMARY OF EMBODIMENTS OF THE INVENTION

Accordingly, in order to provide greater cooling flexibility, reduceoverall cooling costs and provide for runtime localized cooling in oneembodiment, an apparatus includes a plurality of thermal electric cooler(TEC) elements formed in a grid. First control logic dynamically variesa supply current for a first TEC element in the grid according to afirst change in first power density associated with a first area cooledby the first TEC. Second control logic dynamically varies a secondsupply current for a second TEC element in the grid according to asecond change in second power density associated with a second areacooled by the second TEC.

In another embodiment, a method includes determining a first supplycurrent for a first thermal electric cooler (TEC) element in a TEC gridof TEC elements according to a first change in power density associatedwith a first area cooled by the first TEC element. The first supplycurrent is supplied to the first TEC element to control cooling of thefirst TEC element. A second supply current is determined for a secondTEC element in TEC grid according to a second change in power densityassociated with a second area cooled by the second TEC element. Thesecond supply current is supplied to the second TEC element in the TECgrid to control cooling of the second TEC element.

In another embodiment, an apparatus includes a plurality of thermalelectric cooler (TEC) elements formed in a TEC grid. Control logic isconfigured to dynamically vary a supply current supplied to one TECelement in the TEC grid according to a change in power densityrespectively associated with an area cooled by the one TEC element. Thechange in power density is based, in part, on a size of an area beingcooled by the one TEC element. The control logic is configured togenerate a steady state temperature based on the change in power densityand to compare the steady stage temperature to a reference temperatureand generate an error signal indicative thereof and the supply currentis varied according to the error signal.

BRIEF DESCRIPTION OF THE DRAWINGS

The present invention may be better understood, and its numerousobjects, features, and advantages made apparent to those skilled in theart by referencing the accompanying drawings.

FIG. 1 illustrates the basic functioning of a TEC.

FIG. 2 illustrates a TEC with three p-type and n-type thermal electriccouples that are electrically in series and thermally in parallel.

FIG. 3 illustrates the cooling achieved with an SLC and a TEC formedwith bulk silicon.

FIG. 4A illustrates a grid of SLC coolers according to an embodiment.

FIG. 4B illustrates clustering of SLC grid squares.

FIG. 5 illustrates physical clustering of two SLC squares.

FIG. 6 illustrates a high level block diagram of the control structurefor an SLC square.

FIG. 7 illustrates additional details of an embodiment of a feedbackcontrol structure for each SLC square.

FIG. 8 illustrates an embodiment in which a change in power density isprovided at the granularity of a core in a multi-core die.

FIG. 9 illustrates an embodiment in which a processor core provides thechange in power density for various modules within the core.

FIG. 10 illustrates a high level block diagram of an embodiment of acooling system utilizing an SLC array.

FIG. 11 illustrates an embodiment in which the cooling structure and SLCarray are formed as an integral unit.

FIG. 12 illustrates an embodiment in which the SLC array and the die areformed as an integral unit.

FIG. 13A is a top view of a system using a silicon interposer, aplurality of die, and an SLC array to cool the die.

FIG. 13B is a side view of a system using a silicon interposer, aplurality of die, and an SLC array to cool the die.

FIG. 14A is a top view of a system using a silicon interposer, aplurality of die, and an SLC array to cool the die.

FIG. 14B is a side view of a system using a silicon interposer, aplurality of die, and an SLC array to cool the die.

FIG. 15 illustrates power utilization of an integrated circuit under afirst set of conditions.

FIG. 16 illustrates power utilization of an integrated circuit under asecond set of conditions.

The use of the same reference symbols in different drawings indicatessimilar or identical items.

DETAILED DESCRIPTION

Modern processors can generate average heat fluxes in the range of 100W/cm². However, localized hot-spot heat fluxes can be significantlyhigher, e.g., up-to 800 W/cm². Such spatially variable heat fluxes havedrastic differences in temperature (˜30° C.). Traditional heat sinkdesign with air and liquid cooling fail to effectively cool isolatedhot-spots. Solid state cooling using an array of thin-filmthermoelectric coolers (TECs) known as SuperLattice Coolers (SLCs) areeffective at cooling local hot spots. Using principles from feedbackcontrol theory, embodiments herein provide a dynamic thermal managementsolution based on solid-state cooling. Embodiments combine solid statecooling with other cooling techniques. Distributed feedback controllerscontrol the supply current to a grid of SLCs to regulate the temperatureof specific regions, for example, well known hot-spots or spatially andtemporally varying hot-spots.

Since temperature rise time and steady state value at a given locationis influenced largely by only the change in local power density,targeted localized cooling, governed by formal control theoreticprinciples (i) saves energy spent in cooling, (ii) improves performance,(iii) effectively utilizes thermal headroom and (iv) improves devicereliability. SLCs themselves are based on silicon and germanium.Therefore, SLCs can be formed onto the backside of heat sinks such asmicrochannel liquid cooling heat sinks, thus making use of SLCs costeffective as well. Alternatively, since SLCs are based on Si/SiGelayers, certain embodiments integrate the SLCs directly onto the silicondie during manufacturing. In other embodiments, SLCs are formedseparately from the die and heat sinks. Actuating a particular SLCrequires just two terminals (for input current and output current),allowing for a relatively simple control structure.

Improved cooling approaches for graphics processing units (GPUs),central processing units (CPUs), and accelerated processing units(APUs), particularly in the server market, can not only increaseperformance for data center applications but can also reduce the cost ofprovisioning data centers due to reduced cooling requirements.

FIG. 1 illustrates the basic functioning of a TEC 100. A TEC is asolid-state active heat pump that moves heat from the cold side to thehot side of the TEC when an electric current flows between the cold sideand the hot side. The TEC 100 includes a p-type semiconductor portion101 and an n-type semiconductor portion 103 coupled in parallelthermally and in series electrically. The current is applied to the coldside at electrical contact 105 and flows as p-type carriers (holes) fromthe cold side 107 (T_(C)) to the hot side (T_(H)) 109. Current flowsacross the electrical contact 111 electrically connecting the hot sides109 of the p-type portion 101 and the n-type portion 103. In the n-typeportion 103 carriers (electrons) flow from the cold side 107 to the hotside 109, while current flows in the opposite direction from the hotside 109 towards the contact 115. In that way heat is moved from thecold side to the hot side where the heat is dissipated. The device to becooled is thermally coupled to the cold side (and electricallyinsulated) and the heat at the hot side is dissipated using, e.g., aconventional heat sink, to remove heat from the hot side 109.

FIG. 2 illustrates a TEC with three parallel p-type and n-type thermalelectric couples 201, 203, and 205 serially coupled electrically and inparallel thermally to move heat from the cool side 207 to the hot side209. The number of thermal electric couples utilized in a particularimplementation depends on the cooling needs and control structure forthe particular implementation. Instead of using bulk silicon for theTEC, embodiments utilize SLC structures that increase the magnitude ofcooling of the TEC. SLCs have multiple Si and SiGe layers. FIG. 3compares SLC cooling 301 with cooling 303 achieved using a bulk siliconTEC. Note that the cooling efficiency starts to decrease for the SLC andthe TEC formed with bulk silicon after a certain value of supply currentdue to Joule heating. Embodiments herein use TECs formed using bulksilicon or SLCs based on the needs of the particular implementation.

Referring to FIG. 4A, embodiments herein utilize a closed-loopaccurately controlled mechanism to fully exploit the potential of SLCs.Instead of placing SLCs over pre-determined hotspots, embodiments use agrid (also referred to herein as an array) 400 of SLCs formed byindividually controllable SLC grid elements or squares, one of which islabeled 401. Note that the use of the term square is for convenience anddoes not imply that the shape of a grid element is necessarily square. Acontroller dynamically varies the supply current 402 to each squareduring runtime based on a control function that varies the supplycurrent based on local cooling requirements. The control signal forsquare 401 is represented by the variable resistance 403. Thus, ratherthan supplying a current that is either on or off, the control structurevaries the current according to the cooling needs of the area beingcooled as explained further herein. The more current that is supplied tothe SLC square, the greater the cooling supplied by the SLC squarealthough FIG. 3 illustrates there are limits to the amount of currentsupplied above which increased cooling is no longer achieved. One SLCsquare (or group of squares) can receive maximum cooling while othersquares are supplied lower currents or are completely off. Embodimentsutilize SLC grids with different granularities in terms of the numberand size of the SLC squares forming the grid. The SLC grid is designedaccording to, e.g., the physical hot spot size, the size of thecomponent, and/or the size of silicon areas that are most sensitive tohigh temperature. Thus, each square can be smaller than 1 mm² or aslarge as the entire die area, thus, hundreds of mm². The number ofsquares can vary from a few squares to 16, 64, or any suitable numberbased on the requirements of the application.

In embodiments, rather than controlling a single square, the controllogic controls clusters of SLC squares as shown in FIG. 4B. Inembodiments the clusters are logical clusters in that the same currentis supplied to each square 401 and 405 by the controller. The clusteredsquares may be adjacent as shown by squares 401 and 405 butnon-contiguous SLC squares may also be clustered, particularly inlogical clusters. Thus, SLC square 407 may be clustered with one or bothof SLC squares 401 and 405.

In embodiments, rather than being logically clustered, the clusters arephysically linked. FIG. 5 illustrates an embodiment in which switch 501selectively links SLC1 503 and SLC2 505. If the switch 501 is closed,the SLC squares 503 and 505 are serially linked and the current 507supplied to SLC1 square 503 is also supplied to SLC2 square 505. Inembodiments, switch 501 is dynamically controlled during runtimedepending on cooling needs. For example, in embodiments, the largercluster is formed to cool regions of a processor having the same thermalprofile during runtime. While FIG. 5 shows only two squares beingphysically clustered, other embodiments utilize a different number ofclustered SLC squares suitable for the particular application.

Embodiments herein address the problem of cooling spatially andtemporally varying hot-spots using an SLC grid and concepts fromfeedback control theory. A distributed controller framework modulatesthe supply current to elements of an SLC grid structure to regulate thetemperature of different locations of the die to a reference temperaturefor the location. The feedback controllers use local power densityinformation and/or local temperature sensor readings at runtime. FIG. 6illustrates a high level block diagram of the control structure. Thecomponent being cooled 601 supplies the control function 603 with localpower density information and/or local temperature sensor readings. Thecontrol function 603 supplies the control signal 605 (variable supplycurrent) to adjust the cooling provided by TEC 607. That controlstructure is implemented for each SLC square (or cluster of SLC squares)in the grid. The result is lower hot-spot temperatures and less thermalgradient across the die. Therefore, the SLC grid approach leads to amore tightly controlled performance target, a longer silicon lifetime,and potentially lower cost and power at the datacenter level as betterdevice cooling simplifies facility-level cooling complexity.

Embodiments use formal feedback control to vary the supply current ofthe SLC squares individually based on the change in local power density.That provides fine-grain control of the SLC array and allows for greaterdurations of increased processor performance when needed. FIG. 7illustrates an embodiment of the feedback control structure 700 for eachSLC square or SLC cluster. The control structure 700 utilizes a localpower density change (ΔP_(den)) of individual components or subblockswithin plant 701. The local power density change (ΔP_(den)) is based onhardware performance counters (or other mechanisms to provide powerusage reactively or predictively) along with area (mm²) of theindividual components or subblocks within plant 701 being cooled.

Hardware performance counters measure activity in portions of theintegrated circuit. For example, in an embodiment the performancecounter measures how many times a particular signal line in a logicblock toggled. Toggling of that particular signal line indicates use ofthe logic block and therefore corresponds to power consumption. The useof change in power density based on performance counters is predictiveof future temperature increases rather than reactive to an actualincrease. Other embodiments measure power directly from the voltageregulators or measure temperature directly. However, measuringtemperature is problematic in that the change in cooling is reactive tothe change in temperature rather than predictive. Additionally,temperature measurements may not accurately reflect hot spots in theintegrated circuit. In still other embodiments, a power utilizationprediction is based on prior power utilization patterns. In a simpleexample, if a floating point unit was used in a pattern of 100 μs on and100 μs off, the change in power density power assumes that patterncontinues. Thus, while reactive cooling is used in some embodiments ofthe SLC cooling, other embodiments that utilize predictive cooling basedon ΔP_(den) using performance counters or other predictive measuresprovide lower average temperatures and therefore potentially greaterperformance and lower data center cooling costs.

The plant 701 corresponds to the component being cooled and the SLCarray and supplies the change in power density information (ΔP_(den)).Embodiments use performance counters or other predictive mechanisms togenerate ΔP_(den). The feedback gain block 703 generates a steady statetemperature Tss 704 based on the change in power density ΔP_(den) 702supplied by the plant 701. The particular gain used in gain block 703depends on such factors as the location being cooled. The steady statetemperature represents the anticipated steady state temperature afterthe transient increase in temperature is complete without additionalcooling. Note that the change in power density and the steady statetemperature are local in the spatial sense. That is, the change in powerdensity is associated with a particular integrated circuit (or portionthereof) being cooled by one (or a group) of the SLC squares.

Modern processors have the capability to measure energy consumptionper-core thereby allowing for fairly accurate measurement ofinstantaneous power. For example, referring to FIG. 8, a die 800provides a change in power density for each of the processor cores C0,C1, C2, and C3 along with the cache. Coupled with the dimension of thecore area, local power density and changes in power density can bedetermined. Referring to FIG. 9, in an embodiment a core provides thechange in power density with more granularity, e.g., a change in powerdensity is provided for each module in a processor core such as the L1cache, the integer unit (INT), the floating point unit (FPU), the frontend (FRT) and the scheduler (SCH).

Referring back to FIG. 7, a summing circuit 705 compares the steadystate temperature Tss 704 to a reference temperature Tref and generatesan error term 706. The reference temperature is based on a single valuefor the entire die area or is based on a particular physical location ofthe area being cooled. Some portions of the die, e.g., where the computeunits are located, can be expected to run hotter than certain peripheralregions of the die. The error term 706 is supplied to a controller 707.In an embodiment the controller 707 is a fixed gain proportionalintegral derivative (PID) controller or an adaptive gain controller or arobust controller that guarantees a particular output. The controller707 supplies a control term μ 708 that represents the supply current fora particular SLC square (or cluster of SLC squares). Each of the SLCsquares or clusters in the grid have separate control. Given therelative slow nature of temperature excursions, the same controlhardware and software can be time multiplexed to provide the appropriatecontrol functionality for each SLC square or cluster.

In embodiments, the control functionality described in FIG. 7 and shownin FIG. 6 as control 603 is implemented in a programmed processor. Inembodiments, the control functionality is implemented as a softwaredriver that runs on one of the central processing units of a processorbeing cooled by the SLC array. The driver software is stored in volatileand/or non-volatile memory and is available on boot up of the system. Inother embodiments, a programmed firmware controller located in the diethat includes the main processor(s) provides the control functionalitydescribed in FIG. 7 and utilizes firmware stored in a location in thecomputer system that is accessible to the firmware controller thatimplements the control functionality. In still other embodiments, thecontrol functionality is implemented in a management controllerprocessor located outside of the main processor(s) die, e.g., in abaseboard management controller (BMC). In such an embodiment, thesoftware is stored in memory in the computer system, e.g., on the maincircuit board, such that it is available to the BMC. The softwarerequired to implement the control functionality in the variousembodiments described above is stored in non-transitory memory in thecomputer system, e.g., in non-volatile memory or other types ofnon-transitory memory.

FIG. 10 shows a high level block diagram of an embodiment of a coolingsystem utilizing an SLC array 1001. The die 1003 (one or more die) aredirectly cooled by the SLC array and a heat sink 1005 that is thermallycoupled to the SLC array 1001 removes heat from the hot side of the SLCarray. While the SLCs squares in the array provide targeted cooling forspatially and temporally varying hotspots for the die, the cooling ontop of the SLC grid effectively transports the heat away from the SLCarray. The cooling may utilize water (or other liquid) for transportingthe heat away from the SLC grid or a conventional heat sink with fins toallow the heat to be transported away using air, or another heatremoving mechanism. Thus, embodiments utilize a hybrid of solid-stateand an additional cooling mechanism, the additional cooling mechanismbeing used to transport heat away from the SLC array.

In embodiments, the SLC array or grid structure 1001 is formedseparately from the water/air cooling structure 1005 and separately fromthe die 1003. Referring to FIG. 11, in other embodiments the SLC array1101 and the cooling structure 1105, are formed as an integral unit. Inan embodiment with the SLC array integral with the cooling structure,the supply current connections for each of the SLC squares are externalto the die. The simple control structure for controlling programmablecurrent sources is external to the die. In still other embodiments, asshown in FIG. 12, the SLC array 1201 and the die 1203 are formed as anintegral unit. In such a case, in embodiments the programmable currentsources reside in the die and the die supplies the current directly tothe SLC array through vias connecting the die and the SLC array (notshown in FIG. 12). In other embodiments where the SLC array is integralwith the die, the programmable current sources remain external to thedie.

The SLC grid structure can be particularly useful for designs thatutilize a silicon interposer and variable integrated circuits coupled tothe silicon interposer based on a target market. A silicon interposercan achieve faster communication between die. Referring to FIG. 13A atop view shows a silicon interposer 1301 coupled to a plurality ofintegrated circuit die 1303 and an SLC grid 1305 thermally coupled tothe integrated circuit die 1303. FIG. 13B shows a side view of thesystem showing silicon interposer 1301, die 1303, and SLC grid 1305. Theintegrated circuit die may include various types of processors, memory,and other integrated circuits that are advantageously coupled to thesilicon interposer.

FIG. 14A illustrates a design that also includes interposer 1401. Thesame SLC grid 1305 is utilized but different types and/or quantity ofdie are coupled to the interposer. Referring to FIG. 14A a top viewshows a silicon interposer 1401 coupled to a plurality of integratedcircuit die 1403 and the SLC grid 1305 thermally coupled to theintegrated circuit die 1403. FIG. 14B shows a side view of the systemshowing silicon interposer 1401, die 1403, and SLC grid 1305. Theintegrated circuit die 1403 may include various types of processors,memory, and other integrated circuits that are advantageously coupled tothe silicon interposer. In embodiments, the die 1403 are identical dieto the die 1303 but omit several of the die for a different targetmarket. In other embodiments, die 1403 differ from die 1303. However,the same SLC grid array is used for system 1300 and 1400 because thoseSLC squares shown at 1407 are turned off or supplied with a relativelylow current as compared to those SLC squares above active integratedcircuit. Thus, a separate SLC grid does not need to be designed for adifferent configuration of die on an interposer. Instead, the finegrained control of the individual squares allows the grid structure tobe applied to a wide variety of designs and software control can beeffectively utilized to adjust control of the grid for each designenvironment.

FIGS. 15 and 16 illustrate power utilization for an integrated circuitunder changing conditions. FIG. 15 illustrates a first powerutilization. Most of the heat is generated in the center portion of thedie where all the compute units are concentrated in both scenarios. InFIG. 16, additional power is being consumed. While locations 1 and 4 donot change their local power density, between locations 2, 3 and 5 sharethe extra power. The change in power density in the descending order is3>2>5. The area 3 changes because, e.g., an input/output area or otherfunctionality of the integrated circuit was turned on. Note that hotspot temperature depends strongly on local power density and changesrapidly. Using power density changes based on performance counters thatidentifies when the area around location 3 goes from no use to beingused allows proactively cooling location 3 before the location heatsabove the reference voltage thereby providing a cooling solution thatquickly adapts to changing conditions. Temperature dynamics are muchslower as compared to power. Therefore, by the time a control algorithm“reacts” to the rise in temperature, it could be too late to prevent atemperature increase above the target temperature thereby potentiallyadversely affecting other operations in the integrated circuit due toloss of thermal budget. Rapidly estimating the steady state temperaturebased on local power density changes allows a control algorithm topreemptively take corrective action. Equipped with the information aboutlocal steady state temperature, a more targeted cooling technique usingan SLC grid with individualized dynamic current control for grid squaresprovides improved cooling for any silicon die which requires heat to beremoved for normal operation. Traditional heat sink designs are eitherconsiderably slower or simply incapable of addressing such changingconditions.

Thus, embodiments have been described that utilize an SLC grid withindividualized dynamic current control for grid squares. While thedescription has been focused on TECs formed as SLCs, the embodimentsdescribed herein can also be utilized with TECs formed with bulksilicon. The description of the invention set forth herein isillustrative, and is not intended to limit the scope of the invention asset forth in the following claims. Variations and modifications of theembodiments disclosed herein, may be made based on the description setforth herein, without departing from the scope of the invention as setforth in the following claims.

What is claimed is:
 1. An apparatus comprising: a plurality of thermalelectric cooler (TEC) elements formed in a grid; first control logic todynamically vary a first supply current for a first TEC element in thegrid according to a first change in first power density associated witha first area cooled by the first TEC element; and second control logicto dynamically vary a second supply current for a second TEC element inthe grid according to a second change in second power density associatedwith a second area cooled by the second TEC element.
 2. The apparatus asrecited in claim 1, wherein the first change in first power density isbased on at least one performance counter associated with the firstarea; and wherein the second change in second power density is based onat least another performance counter associated with the second area. 3.The apparatus as recited in claim 1, wherein the first change in firstpower density is further based on a size of the first area; and whereinthe second change in second power density is further based on a size ofthe second area.
 4. The apparatus as recited in claim 1, wherein thefirst control logic is configured to generate a first steady statetemperature for the first area based on the first change in first powerdensity, compare the steady state temperature to a first referencetemperature and generate a first error signal indicative of the compare,and adjust the first supply current based on the first error signal; andwherein the second control logic is configured to generate a secondsteady state temperature for the second area based on the second changein second power density, compare the second steady state temperature toa second reference temperature and generate a second error signalindicative of the compare, and adjust the second supply current based onthe second error signal.
 5. The apparatus as recited in claim 4 whereinthe first reference temperature and the second reference temperature aredetermined according to, respectively, a location of the first area andthe second area.
 6. The apparatus as recited in claim 1 wherein at leasttwo TEC elements in the grid are controlled by a single supply currentand at least another element in the grid is controlled by a differentsupply current.
 7. The apparatus as recited in claim 6 furthercomprising at least one electrical connection controllable duringruntime coupling the at least two TEC elements to provide a path for thesingle supply current to flow from one of the at least two TEC elementsto another of the at least two TEC elements.
 8. The apparatus as recitedin claim 1 further comprising: a first silicon die disposed under afirst one or more TEC elements of the grid; a second silicon diedisposed under a second one or more TEC elements of the grid; and athird one or more TEC elements of the grid disposed above an absence ofactive silicon die.
 9. The apparatus as recited in claim 1 furthercomprising: one or more integrated circuit die; the grid thermallycoupled to the one or more integrated circuit die; and another coolingstructure thermally coupled to the TEC grid to remove heat from the TECgrid.
 10. A method comprising: determining a first supply current for afirst thermal electric cooler (TEC) element in a TEC grid of TECelements according to a first change in power density associated with afirst area cooled by the first TEC element; supplying the first supplycurrent to the first TEC element to control cooling of the first TECelement; determining a second supply current for a second TEC element inthe TEC grid according to a second change in power density associatedwith a second area cooled by the second TEC element; and supplying thesecond supply current to the second TEC element to control cooling ofthe second TEC element.
 11. The method as recited in claim 10, furthercomprising” determining the first change in power density, at least inpart, according to a first performance counter associated with the firstarea and a first size of the first area; and determining the secondchange in power density, at least in part, according to a secondperformance counter associated with the second area and a second size ofthe second area.
 12. The method as recited in claim 11 furthercomprising: generating a first steady state temperature based on thefirst change in power density; generating a first error signal based oncomparing the first steady state temperature to a first referencetemperature; generating the first supply current based, at least inpart, on the first error signal; generating a second steady statetemperature based on the second change in power density; generating asecond error signal based on comparing the second steady statetemperature to a second reference temperature; and generating the secondsupply current based, at least in part, on the second error signal. 13.The method as recited in claim 12 wherein the first referencetemperature is the same as the second reference temperature.
 14. Themethod as recited in claim 12 wherein the first reference temperaturecorresponds to the first area and the second reference temperaturecorresponds to the second area and the first and second referencetemperatures are different.
 15. The method as recited in claim 10further comprising: supplying the first supply current from the firstTEC element to a third TEC element electrically coupled to the first TECelement.
 16. The method as recited in claim 10 further comprisingdynamically controlling at least one controllable electrical connectioncoupling at least two TEC elements in the TEC grid to allow a supplycurrent to flow from a first TEC element of the two TEC elements to asecond TEC element of the two TEC elements.
 17. The method as recited inclaim 10 further comprising actively controlling a first plurality ofTECs in the TEC grid and not actively controlling at least one TECelement in the TEC grid and keeping the at least one TEC element in theTEC grid in a permanent off state.
 18. The method as recited in claim 10further comprising: removing heat from the TEC grid using liquidcooling.
 19. The method as recited in claim 10 further comprising:logically grouping a first plurality of TEC elements in the TEC grid bysupplying identical supply currents to each of the first plurality ofTEC elements and supplying a different supply current to at leastanother of the TEC elements in the TEC grid.
 20. An apparatuscomprising: a plurality of thermal electric cooler (TEC) elements formedin a TEC grid; control logic configured to dynamically vary a supplycurrent supplied to one TEC element in the TEC grid according to achange in power density respectively associated with an area cooled bythe one TEC element; wherein the change in power density is based, inpart, on a size of an area being cooled by the one TEC element; whereinthe control logic is configured to generate a steady state temperaturebased on the change in power density and to compare the steady statetemperature to a reference temperature and generate an error signalindicative thereof; and wherein the supply current is varied accordingto the error signal.